ML Report Descriptions:
The two main goals with this document are to provide a wide range of output to investigate high level performance and insights, and to deliver a high quality report design layout to increase user experience. The metrics provided are intended to be semi-comprehensive. One can always dig deeper into results to gain further insights. In light of that, the results are intended to provide information that one can come to a reasonable conclusion about their model or to find the areas where they need to dig a little deeper.
Evaluation Metrics
This section contains statistics and variable importance measures to help the user understand model performance at a high level. Train Data results are included and can be used to compare against the Test Data results to identify over / under fitting of models.
Evaluation Plots
This section contains visualizations that span the range of predicted values and the associated accuracies across that range. The predicted values range is broken up into every 5th percentile to provide a wide range for evaluation.
Model Interpretation sections
This section contains visualizations intended to open up the black box of your algorithm. When one inspects coefficients from a regression model, the insights they gain are two-fold: - get an understanding about statistics significance - gain an understanding of the variable’s effect on the target variable However, not all relationships are linear and sometimes the user doesn’t specifiy an appropriate model structure to fully capture the nature of the relationship, which can lead to incorrect conclusions about both statistical signifance and the nature of the relationship. These visualizations provide a way to understand what the exact nature of those relationships are (in a visual manner) and if the user chooses, they can attempt to fit the relationship more precisely with an appropriate statistical model in order to gain a better understanding of statistical significance.
Evaluation Metrics
Expand
MultiClass Metrics Tables
MultiClass Metrics
Binary Metrics 1 vs All Tables
Model Metrics Tables
TestData
Performance Metrics
## [1] "TestData_I"
## [1] "TestData_J"
## [1] "TestData_O"
## [1] "TestData_U"
## [1] "TestData_Y"TrainData + ValidationData
Performance Metrics
## [1] "TrainData_I"
## [1] "TrainData_J"
## [1] "TrainData_O"
## [1] "TrainData_U"
## [1] "TrainData_Y"Variable Importance Table
Variable Importance
Interaction Importance Table
Interaction Importance
## [1] "Interaction importance is only available with CatBoost"
Evaluation Plots
Expand
Variable Importance Plots
Expand
Calibration Plots
Expand
TestData
Calibration Plot
TrainData + ValidationData
Calibration Plot
ROC Plots
Expand
TestData
ROC Plots
TrainData + ValidationData
Expand
Lift & Gains Plots
Expand
TestData
Lift & Gains Plots
TrainData + ValidationData
Expand
Model Interpretation
Expand
Partial Dependence Plots:
Numeric-Features
Expand
Partial Dependence Line Plots
Expand
TestData
Partital Dependence Line Plots
TrainData + ValidationData
Partital Dependence Line Plots
Partial Partial Dependence Plots:
Categorical-Features
Expand
TestData
Partital Dependence Bar Plots
TrainData + ValidationData
Partital Dependence Bar Plots
Model MetaData
Expand
Parameters and Settings
Model Parameters
## [1] "HasTime: FALSE"
## [1] "EvalMetric: "
## [1] "LossFunction: MultiClassOneVsAll"
## [1] "OutputSelection: Importances" "OutputSelection: EvalPlots"
## [3] "OutputSelection: EvalMetrics" "OutputSelection: Score_TrainData"
## [1] "TargetColumnName: Adrian"
## [1] "FeatureColNames: Independent_Variable1"
## [2] "FeatureColNames: Independent_Variable2"
## [3] "FeatureColNames: Independent_Variable3"
## [4] "FeatureColNames: Independent_Variable4"
## [5] "FeatureColNames: Independent_Variable5"
## [6] "FeatureColNames: Independent_Variable6"
## [7] "FeatureColNames: Independent_Variable7"
## [8] "FeatureColNames: Independent_Variable8"
## [9] "FeatureColNames: Independent_Variable9"
## [10] "FeatureColNames: Independent_Variable10"
## [11] "FeatureColNames: Factor_2"
## [1] "PrimaryDateColumn: "
## [1] "WeightsColumnName: "
## [1] "IDcols: IDcol_1" "IDcols: IDcol_2"
## [1] "EncodeMethod: catboost"
## [1] "TrainOnFull: FALSE"
## [1] "task_type: GPU"
## [1] "NumGPUs: 0"
## [1] "DebugMode: FALSE"
## [1] "ReturnModelObjects: TRUE"
## [1] "SaveModelObjects: FALSE"
## [1] "ModelID: Test_Model_1"
## [1] "model_path: C:/Users/Bizon/Documents/GitHub/AutoQuant"
## [1] "metadata_path: C:/Users/Bizon/Documents/GitHub/AutoQuant"
## [1] "ClassWeights: "
## [1] "NumOfParDepPlots: 11"
## [1] "eval_metric: MultiClassOneVsAll"
## [1] "loss_function: MultiClassOneVsAll"
## [1] "grid_eval_metric: Accuracy"
## [1] "BaselineComparison: default"
## [1] "MetricPeriods: 10"
## [1] "PassInGrid: "
## [1] "GridTune: FALSE"
## [1] "MaxModelsInGrid: 30"
## [1] "MaxRunsWithoutNewWinner: 20"
## [1] "MaxRunMinutes: 1440"
## [1] "Trees: 50"
## [1] "Depth: 6"
## [1] "LearningRate: "
## [1] "L2_Leaf_Reg: "
## [1] "RandomStrength: 1"
## [1] "BorderCount: 128"
## [1] "RSM: "
## [1] "BootStrapType: Bayesian"
## [1] "GrowPolicy: SymmetricTree"
## [1] "langevin: FALSE"
## [1] "diffusion_temperature: 10000"
## [1] "model_size_reg: 0.5"
## [1] "feature_border_type: GreedyLogSum"
## [1] "sampling_unit: Object"
## [1] "subsample: "
## [1] "score_function: Cosine"
## [1] "min_data_in_leaf: 1"
## [1] "Algo: CatBoost"
## [1] "TargetType: MultiClass"
## [1] "PredictionColumnName: Predict"
## [1] "TargetLevels: c(\"I\", \"J\", \"O\", \"U\", \"Y\")"
## [2] "TargetLevels: 1:5"Grid Tuning Metrics
Grid Tuning Metrics
## [1] "GridTuning was not conducted"